Image reranking and presentation for visual exploration

ABSTRACT

The present disclosure relates to systems, devices, and methods for a visual exploration of search engine results. The systems, devices, and methods may present, in response to receiving the search engine results for a query to perform a task, a search result webpage with a cascading waterfall layout that includes a plurality of images from a plurality of webpages included in the search results. The search result webpage may also present, adjacent to the plurality of images, structured data corresponding to a selected image from the plurality of images where the structured data includes details from the webpage to complete the task.

BACKGROUND

Users may use search engines for exploration to find inspirations andnew ideas and to learn concepts to formulate good queries. Sometimessuch explorations have a strong visual component (e.g., the user maywant to know how to decorate a backyard herself). Typically, the firststep is to find such ideas that are attractive and suitable for theneeds of the user. Therefore, images are crucial to this type ofexploration. However, only images are not sufficient because the nextsteps typically are to learn how to achieve this decoration, whatmaterials are needed, how long the project takes, and/or the price forthe project.

BRIEF SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

This present disclosure addresses problems with current search enginesthat return images disconnected from structured data, text searchresults without any images, or results lacking sufficient details for auser to complete a desired task. For example, a search engine userwishing to make a lasagna dish may want to begin by looking for alasagna image that meets the user's criteria. In this example, the userdoes not want to see images of lasagna dishes that don't have associatedingredient lists and recipe steps. Nor does the user want a list oflasagna recipes without any images to show the final product. As anexample, the user enters a query, such as “lasagna recipe.” The presentsystem obtains an index that includes documents (e.g., webpages) thatcontain both a lasagna image and sufficient details to recreate thedish. In this example, the system then presents a search results pageswith lasagna images from the index (e.g., in a waterfall layout) inresponse to a user query. If the user selects an image in this example,a corresponding details card is displayed with recipe details (e.g.,ingredients). If the user does not approve of the ingredients for thatrecipe, the user may select a new image and the first details card isthen hidden and a new details card for the new image is displayed withinthe search page. In this way the user may quickly and efficiently find arecipe with a desired final appearance and desired ingredients.

Some embodiments include a method for presenting structured data withsearch engine results on a display of a device. The method may includereceiving a query from a user, wherein the query identifies a multi-steptask and comprises a request for task details for completing the task.The method may include providing the query to a search engine. Themethod may include responsive to providing the query, receiving resultsfrom the search engine, wherein the received results includeidentification of a plurality of webpages, wherein each webpage of theplurality of webpages includes details to complete the task. The methodmay include presenting on the display a search result webpage responsiveto receiving the results, the search result webpage including aplurality of images from the plurality of webpages, wherein theplurality of images are presented in a cascading waterfall layout thatincludes a plurality of columns. The method may include receiving a userselection of an image from the plurality of images; and responsive tothe user selection: presenting, adjacent to the plurality of images onthe search result webpage, structured data corresponding to the image,the structured data comprising details from the webpage to complete thetask; and adjusting the presentation of the cascading waterfall layoutbased on the selection of the image, wherein the adjusting includes atleast one of: resizing a column of the plurality of columns, andrearranging an ordering of images within at least one column of theplurality of columns.

Some embodiments include a system or device with one or more processors;memory in electronic communication with the one or more processors; andinstructions stored in the memory, the instructions being executable bythe one or more processors to perform any of the methods disclosedherein.

Some embodiments include a computer-readable medium storing instructionsexecutable by a computer system or device. The computer-readable mediumincludes instructions for causing the computer system or device toperform any of the methods described herein.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the disclosure may be realized and obtained by means ofthe instruments and combinations particularly pointed out in theappended claims. Features of the present disclosure will become morefully apparent from the following description and appended claims or maybe learned by the practice of the disclosure as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otherfeatures of the disclosure can be obtained, a more particulardescription will be rendered by reference to specific embodimentsthereof which are illustrated in the appended drawings. For betterunderstanding, the like elements have been designated by like referencenumbers throughout the various accompanying figures. While some of thedrawings may be schematic or exaggerated representations of concepts, atleast some of the drawings may be drawn to scale. Understanding that thedrawings depict some example embodiments, the embodiments will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates an example environment for presenting search engineresults for a search engine query in accordance with some embodiments ofthe present disclosure.

FIG. 2 illustrates an example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images for ado it yourself (DIY) project and structured data for a selected image inaccordance with some embodiments of the of the present disclosure.

FIG. 3 illustrates another example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images for aDIY project and structured data for a selected image in accordance withsome embodiments of the present disclosure.

FIG. 4 illustrates an example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images forrecipes and search results in accordance with some embodiments of thepresent disclosure.

FIG. 5 illustrates an example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images forrecipes in accordance with some embodiments of the present disclosure.

FIGS. 6A and 6B illustrate an example graphical user interface of awebpage displaying a cascading waterfall layout with a plurality ofimages for recipes and structured data for a selected image inaccordance with some embodiments of the present disclosure.

FIG. 7 illustrates an example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images forrecipes and structured data for another selected image in accordancewith some embodiments of the present disclosure.

FIG. 8 illustrates an example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images forrecipes and one or more filters in accordance with some embodiments ofthe present disclosure.

FIG. 9 illustrates an example graphical user interface of a webpagedisplaying a cascading waterfall layout with a plurality of images andvideos for recipes in accordance with some embodiments of the presentdisclosure.

FIG. 10 illustrates an example method for presenting structured datawith search engine results in accordance with some embodiments of thepresent disclosure.

DETAILED DESCRIPTION

This disclosure generally relates to presenting search engine results.Users may use search engines for exploration to find inspirations andnew ideas and to learn concepts to formulate good queries. Sometimessuch explorations have a strong visual component (e.g., the user maywant to know how to decorate a backyard herself). Typically, the firststep in visual exploration is to find such ideas that are attractive andsuitable for the needs of the user. Therefore, images are crucial tothis type of visual exploration. However, only images are not sufficientbecause the next steps typically are to learn how to achieve thisdecoration, what materials are needed, how long the project takes,and/or the price for the project. Users may use visual explorationsearches to find inspirations and new ideas. For example, users usevisual exploration searches when finding recipes, do it yourself (DIY)projects, and/or making travel plans.

Existing techniques do not support exploratory image searches. Existingtechniques rank images without taking into account possible ideas and/orprojects behind those images and what context is important for the ideasand/or projects. As such, exploration and comparison between differentideas and/or projects is difficult using existing techniques. Userseither constantly need to switch between image searching (to get overallunderstanding of the idea and/or the project), source webpages (tounderstand needed materials, ingredients, and/or time), and web searches(to search for similar ideas based on terms found on source pages or tolearn about those materials/ingredients and other specifics of ideas).Alternatively, users must choose one way of information presentation(either images or texts), and thus, reducing effectiveness of theexploration.

This disclosure provides methods and devices for providing bettercontext for comparing ideas (recipes, travel destinations, DIY projects,etc.) by showing not only images, gifs, and/or videos for search resultsfor a task, but also relevant page information using structured dataextracted from the webpage associated with the images. The structureddata may include, for example, a title of the webpage, a uniformresource locator (URL) for the webpage, ingredients, materials, timeneeded for the recipe or the project, and/or rich metadata extractedfrom the webpage. The presentation provided by the methods and devicesmay include a cascading waterfall layout with a plurality of images. Theplurality of images may be associated with a plurality of webpages thatinclude details for completing the task. The plurality of webpages maybe included in search results returned by a search engine in response toa query for the task. The cascading waterfall layout is concise enoughto allow quick comparison and exploration of the ideas. The cascadingwaterfall layout may be presented at a top portion of a search resultwebpage above the general search results returned by the search enginefor a query. This disclosure includes several practical applicationsthat provide benefits and/or solve problems associated with presentingsearch engine results.

The methods and devices may determine to trigger the cascading waterfalllayout of images, videos, and/or gifs based on the query provided by theuser. The methods and devices may use a trained machine learning modelto determine whether to trigger the presentation of the cascadingwaterfall layout. The machine learning model may be trained based onfully supervised data (labeled by humans) or by weakly supervised dataof search engine logs to extract images searches where users areinterested in the source pages, not only in the images, based on whetherthe user clicks on pages more often than the images from the searchengine logs. The user clicking on the source pages may indicate that thequery has visual exploration intent (exploration because of clicking onthe webpages may indicate interest in additional context and/or relatedinformation) and visual intent because the query was issued to an imagesearch, not a web search. If the machine learning model determines thatthe query includes a visual intent, the presentation of the cascadingwaterfall layout of images may be triggered.

The methods and devices may determine which images, videos, and/or gifsto include in the cascading waterfall by using a ranking based on aweighted combination of aspects. For example, the image relevance to thetask, image attractiveness, and/or page usefulness may go into theranking. The results of the ranking may be deduplicated based on imagecontent. For exploratory queries, higher weights may be assigned to pagerelated aspects and idea-based deduplication may be applied. Forexample, if there are multiple images for the same idea of garagedecoration (e.g., just photographed from different angles), only oneimage per idea may be shown or the ideas may be grouped and shown withmultiple images with the same context information. The order of imagespresented in the cascading waterfall may be also specific (e.g., for DIYprojects the order of images presented may be a logical order of stepsto complete the project). The methods and devices may also integrate avisual search where users search by images (e.g., exploring ideas withsimilar look, such as, bedrooms in similar styles, colors) indetermining which images to include in the cascading waterfall layout.

As such, the methods and devices may optimize the search experience forvisual exploration searches by improving the user experiences for weband/or image searches. The methods and devices may be used to visuallyrepresent search engine results by providing a cascading waterfalllayout of images that is concise enough to allow quick visual comparisonand exploration of the ideas presented in the cascading waterfalllayout. The user may quickly browse more ideas by reviewing thestructured data for different images selected from the cascadingwaterfall layout without having to redirect to the webpages associatedwith the selected images. Users may use the methods and devices to makemore accurate and/or more informed decisions for determining whichwebpages to visit provided in a search result.

Referring now to FIG. 1 , illustrated is an example environment 100 forpresenting search engine results. Environment 100 may include one ormore devices 102 in communication with one or more servers 106 via anetwork 104. The one or more devices 102 may be used by users to enterqueries 12 for searching different webpages for a task 14 and/orperforming a visual exploration of one or more tasks 14. For example,the user may enter the queries 12 on a search webpage presented on adisplay 10 in communication with the device 102.

The queries 12 may be visual searches for upcoming projects, cookingactivities, and/or vacations. The queries 12 may also be explorationsearches for new ideas and/or recommendations. The users may providetext describing the tasks 14 in the queries 12. The queries 12 mayidentify a task 14 and/or may include a request for task details forcompleting the task 14. The tasks 14 may be multi-step tasks, such as,but not limited to, DIY projects, cooking, and/or trip planning.

The device 102 may provide the query 12 for the task 14 to a searchengine 26 that searches one or more datastores 108, 110 with a pluralityof webpages 31, 33 in response to the received query 12. The webpages31, 33 may include a plurality of images 32, a plurality of videos 34,and/or a plurality of text 36. The one or more datastores 108, 110 maystore webpages 31, 33 published from the same content provider or maystore webpages 31, 33 published from different content providers. Forexample, webpages 31 from a first content provider are stored indatastore 108 and webpages 33 from the first content provider are storedin datastore 110. Another example may include webpages 31 a firstcontent provider are stored in datastore 108 and webpages from a secondcontent provider are stored in datastore 110. Another example mayinclude webpages 31 from a first content provider and a second contentprovider are stored in datastore 108 and webpages 33 from a thirdcontent provider and a fourth content provider are stored in datastore110.

The search engine 26 may generate search results 24 that identify aplurality of webpages 30 from the datastores 108, 110 that discuss thetask 14, include details to complete the task 14, and/or may be relevantto the task 14. The search engine 26 may also generate textual searchresults 24 with a plurality of search results for the query 12. Thewebpages 30 included in the search results 24 may include a plurality ofimages 32, a plurality of videos, and/or a plurality of text 36.

A query classifier 29 may perform further processing on the searchresults 24 to determine which images 32 and/or which webpages 30 fromthe search results 24 to present for the task 14 in response to thequery 12. The query classifier 29 may be a trained machine learningmodel that analyzes the search results 24 and the query 12 to determinewhich webpages 30 and/or images 32 of the search results 24 to presentfor the task 14.

The query classifier 29 may be trained based on human labeled data or byweakly supervised data for a specific segment or domain for the queryclassifier (e.g., recipes, DIY, travel plans). As such, different queryclassifiers 29 may be selected for analyzing the search results 24 basedon the segment or domain of the task 14. The segment or domain (e.g.,recipes, DIY, travel plans) of the task 14 may be predicted by anotherquery classifier trained on human labeled data. The query classifier 29may be trained based on fully supervised data (labeled by humans) orweakly supervised data for more than one segments and/or domains. Thequery classifier 29 may identify a plurality of images 40 to presentand/or a plurality of webpages 38 to present from the search results 24.As such, a subset of the images 32 and/or the webpages 30 provided inthe search results 24 may be selected for presentation in response tothe query 12.

The query classifier 29 analyzes a quality of the images 32 indetermining whether to include the images 32 in the plurality of images40 to present. The quality of the image 32 may include, for example,attractiveness of the images, and/or the relevance of the image to thequery 12 and/or the task 14. If the quality of the image 32 is above athreshold level, the image 32 is included in the plurality of images 40to present. The query classifier 29 may also analyze a quality of thewebpages 30 associated with the images 32 in determining whether toinclude the images 32 in the plurality of images 40 to present and/orinclude the webpages 30 in the plurality of webpages 38 to present. Thequality of the webpages 30 may include, for example, anauthoritativeness of the webpages 30 and/or a source of the webpages 30.If the quality of the webpages 30 is high (e.g., the webpage is from anapproved source or an authenticated source), the webpage 30 is includedin the plurality of webpages 38 to present. If the quality of thewebpages 30 is low (e.g., the webpage is from an unauthenticatedsource), the webpage 30 is not included in the plurality of webpages 38to present. The query classifier 29 may use both the quality of theimages 32 and the quality of the webpages 30 in determining theplurality of images 40 to display with the search results 24 in responseto the query 12.

The query classifier 29 may group the plurality of images 40 from thesearch results 24 according to ideas, projects, and/or traveldestinations associated with the images 32. Groupings of the pluralityof images 40 may be based on the webpage 38 associated with the images40. In addition, grouping of the plurality of images 40 may be by aspecial aggregation method where if the same idea or the same traveldestination is described in multiple webpages 38, the plurality ofimages 40 may be aggregated into one group. If multiple ideas aredescribed in the same webpage 38, the different ideas may be extractedand multiple groups formed for the plurality of images 40.

The query classifier 29 may extract structured data 22, such as,meta-information related to the grouped images. The structured data 22may include, but is not limited to, recipes, prices, user ratings,reviews, a uniform resource locator (URL) link to the webpage, thewebpage name steps for performing the task 14, step-by-step images forperforming the task 14, an instructional video for performing the task14, a list of materials needed for performing the task 14, an estimatedtime required to complete the task 14, direct links to a portion of thewebpage where the task 14 is described, an aggregate of information fromdifferent webpages for the task 14 (additional images, steps, materialsdescription, etc.), substitute materials for the task 14, and/or localinformation 44 associated with the task 14.

The structured data 22 extraction may be performed on the plurality ofwebpages 38 associated with the plurality of images 40. As such, thestructured data 22 may be extracted from the plurality of webpages 38.In addition, the structured data 22 extraction may occur from sourcesseparate from the plurality of webpages 30. Thus, the structured data 22may be extracted from sources other than the plurality of webpages 30and/or extracted from the plurality of webpages 38. The structured data22 extraction may be performed by rule-based methods (e.g.,pre-specified patterns for specific segments like recipes) or machinelearning based methods based on manually labeled data. In addition, thestructured data 22 extraction may use an existing knowledge base forparticular segments or domains (e.g., recipes, DIY, recipes).

The structured data 22 may include local information 44 associated withthe task 14. Examples of local information 44 include, but are notlimited to, estimated cost of materials for the task 14, localavailability of materials for the task 14, local cost of materials forthe task 14, and/or a store to purchase the materials for the task 14located nearby a location of the device 102. The local information 44may be provided from a third party datastore 112 that stores the localinformation 44. As the local information 44 changes and/or is updated(e.g., availability of materials change or inventory at the storeschanges), the local information 44 may be updated in the third partydatastore 112.

The query classifier 29 may apply one or more rankings 42 to theplurality of webpages 38. The rankings 42 may be used to determine anorder for presenting the plurality of webpages 38 and/or a plurality ofimages 40 associated with the plurality of webpages 38. The rankings 42may be applied to the groups of the plurality of images 40. For example,the query classifier 29 takes a max or sum of clicks for each image inthe plurality of images 40 or takes features of the most dominant imageof the plurality of images 40 if aggregation for a particular feature isnot desirable. The query classifier 29 may append the rankings 42 to theplurality of images 40 and/or the plurality of webpages 38.

The rankings 42 may also take into account the extracted structured data22. If the plurality of webpages 38 have rich, relevant, and/or use casespecific (e.g., having positive user reviews) structured data 22, thenthe rankings 42 for the plurality of webpages 38 may be higher relativeto the plurality of webpages 38 with less structured data 22 or lessrich structured data 22. In some embodiments, the webpages 38 mayinclude a video describing the task details. For example, videos mayinclude instructional videos, such as, a YouTube instructional video.The query classifier 29 may apply a higher ranking 42 to webpages 38with instructional videos for the task 14 as compared to webpages 38with limited instructions for the task 14. The query classifier 29 mayappending feature vectors extracted for the structured data 22 with therankings 42 information.

The search engine 26 may send the search results 24 to the device 102for presentation on the display 10. The search engine 26 may also sendto the device 102 the identified plurality of webpages 38 that includedetails and/or information for completing the task 14. The searchresults 24 may be based on the general search index for the task 14. Theinformation included in the search results 24 may be different from theinformation included in the identified plurality of webpages 38. Forexample, different webpages, videos, and/or articles related to the task14 may be include in the general search results 24 as compared to theplurality of webpages 38. In addition, the information included in thesearch results 24 may be the same as the information included in theplurality of webpages 38. For example, the same webpages may be includedin the search results 24 as the plurality of webpages 38 included in thecascading waterfall layout 18. Another example may include the samevideos 34 included in the search results 24 and the cascading waterfalllayout 18.

The device 102 may have a presentation component 28 that determines apresentation of the received search results 24, the plurality ofwebpages 38 and/or associated plurality of images 40, and/or anyreceived structured data 22 for the plurality of webpages 38.

The presentation component 28 may generate a cascading waterfall layout18 to present the plurality of images 40 for the identified plurality ofwebpages 38. The plurality of images 40 may include, for example,images, gifs, and/or videos. The number of images included in theidentified plurality of images 40 may be a predetermined or presetnumber of images. For example, the cascading waterfall layout 18 mayinclude fifty images for the task 14.

The presentation component 28 may determine to trigger the presentationof the cascading waterfall layout 18 based on the query 12 provided bythe user. The presentation component 28 may use a trained machinelearning model to determine whether the query includes a visual intent.The machine learning model may be trained using fully supervised data(labeled by humans) or weakly supervised data of search engine logs toextract images searches where users are interested in the source pages,not only in the images, based on whether the user clicks on pages moreoften than the image from the search engine logs. The user clicking onthe source pages may indicate that the query has visual explorationintent (exploration because of clicking on the webpages may indicateinterest in additional context and/or related information) and visualintent because the query was issued to an image search, not a websearch. The trained machine learning model may also be trained on imageintent using human labeled data. The query classifier 29 may be trainedto determine an image intent in a query 12 and select whether thepresent the cascading waterfall layout 18 based on the image intent ofthe query 12. The presentation component 28 may use the trained machinelearning model to determine that the query 12 includes a visual aspectand/or is a visual exploration search and may trigger the presentationof the cascading waterfall layout 18.

The cascading waterfall layout 18 may include a plurality of columns 50for presenting the plurality of images 40. The width of the plurality ofcolumns 50 may be based on the image result sizes and/or the displaysize 48 of the device 102. In addition, the width of the plurality ofcolumns 50 may be a fixed size. The number of columns included in theplurality of columns 50 may be based on the display size 48 of thedevice 102. For example, devices 102 with a larger display size 48 mayhave more columns included in the plurality of columns 50 as compared todevices 102 with a smaller display size 48. The number of rows includedin the cascading waterfall layout 18 may be selected based on thedisplay size 48 and/or the size of the images. If the plurality ofimages are landscape-oriented with a small height, more rows may beselected relative to images with a larger height. As such, thepresentation of the cascading waterfall layout 18 may differ betweendevices 102 with different display sizes 48.

The presentation component 28 may place the plurality of images 40 intothe plurality of columns 50 using the rankings 42. The presentationcomponent 28 may also use information about the presentation of theplurality of images 40 in the webpages 38 in determining the order forpresenting the plurality of images 40. For example, a hierarchicalorganization of the webpage 38 may be used to understand which level tokeep an image (e.g., a first image from each section after filtering outirrelevant images or an order of images in a DIY segment). As such, theoriginal order of the images in the webpages 38 may be used indetermining how to present the plurality of images 40.

The presentation component 28 may start with a first image from theplurality of images 40 and place the image into a first column of theplurality of columns 50. A next image from the plurality of images 40 isplaced into a second column of the plurality of columns 50. The imageplacement continues until one image is placed per column. The images mayhave different sizes and the images per column may differ in size.

The presentation component 28 may calculate which column has a leastnumber of pixels from the top of the search result webpage 16 and placesa next image from the plurality of images 40 into the identified column.The presentation component 28 may continue to calculate which column hasa least number of pixels from the top of the search result webpage 16and place a next image into the identified column until a predeterminednumber of images are presented in the cascading waterfall layout 18. Thepredetermined number of images may present an initial subset of theplurality of images 40 to include in the cascading waterfall layout 18.For example, if the plurality of images 40 includes 50 images for thetask 14, the predetermined number for the subset of initial imagespresented may be 20 or 30 images. The predetermined number of images maybe based on the display size 48 of the device 102. Devices 102 withsmaller display sizes 48 may present 20 images in the cascadingwaterfall layout 18 and devices 102 with larger display sizes 48 maypresent 30 images in the cascading waterfall layout 18. As such, thepresentation component 28 may fill up the cascading waterfall layout 18with images based on a combination of the rankings of the images 40 andthe display requirements of the device 102.

The plurality of images 40 initially presented in the cascadingwaterfall layout 18 may be the images with a highest ranking 42. Forexample, if materials mentioned for a DIY task 14 from one of theplurality webpages 38 are not available locally, the associated imagefor that webpage may not be included in the initial subset of images.Another example may include adding an image from a webpage of theplurality of webpages 38 to the plurality of images 40 that hasstep-by-step images for the DIY task 14 described in the webpage.

A user may select to see more and/or less images than the initial imagesdisplayed in the cascading waterfall layout 18. For example, a user mayselect an icon to see more images. The presentation component 28 mayadjust the presentation of the cascading waterfall layout 18 to includemore images from the plurality of images 40. For example, the size ofthe images are resized to include more images presented per row. Anotherexample may include adding more rows of images so a user may scrollthrough the rows of additional images.

A user may also select one image of the plurality of images 40 presentedin the cascading waterfall layout 18. For example, the user may want toreview the image further or obtain additional information for the image.The presentation component 28 may present the structured data 22associated with the webpage for the selected image. The structured data22 may be presented adjacent to the cascading waterfall layout 18. Insome embodiments, the presentation component 28 may present thestructured data using a rich card 20. By presenting the structured data22 next to the cascading waterfall layout 18 on the same search resultwebpage 16, users may make more informed decisions about whether to goto the source webpage for the selected image.

The presentation component 28 adjusts the presentation of the cascadingwaterfall layout 18 based on the selection of the image so that thestructured data 22 is presented next to the cascading waterfall layout18. Adjusting the presentation of the cascading waterfall layout 18 mayinclude, for example, resizing a column of the plurality of columns 50,reducing a number of columns of the plurality of columns 50, rearrangingan ordering of the images within at least one column of the plurality ofcolumns 50, rearranging a position of the images within the plurality ofcolumns 50, and/or resizing the images within the plurality of columns50. As such, the cascading waterfall layout 18 has an adaptivearrangement of columns based on available screen real estate andresponsive design.

The presentation component 28 adjusts the presentation of the cascadingwaterfall layout 18 without issuing a new search request to the searchengine 26 and/or changing the query 12 from the user. If a user selectsa different image from the plurality of images 40, the presentationcomponent 28 may replace the structured data 22 presented with thestructured data 22 for the webpage associated with the different image.The user may view different structure data 22 for different imagesselected from the plurality of images 40 without issuing a new search tothe search engine 26 or changing the query 12 from the user, resultingin an improvement of the search result webpage 16 by not having torefresh the entire search result webpage 16 each time a new image isselected. As such, the user may easily switch between the structureddata 22 for different images selected resulting in quick and easycomparison of the different tasks 14 for the selected images. Moreover,the user may quickly browse more ideas by reviewing the structured data22 for different images selected without having to redirect to thewebpages associated with the selected images.

In some embodiments, the presentation component 28 may presentrecommendations 49 with the structured data 22 based on the selectedimage. The recommendations 49 are optionally included in the same richcard 20 as the structured data 22, or adjacent to it. Therecommendations 49 may include similar content to the content expressedin the selected image and/or the content expressed in the associatedwebpage. The recommendations 49 may not be included in the cascadingwaterfall layout 18 but may be obtained in response to a new queryconstructed based on the selected image.

In some embodiments, the presentation component 28 may present one ormore videos 34 for the task 14 with the plurality of images 40 in thecascading waterfall layout 18. The videos 34 may describe task details,such as, but not limited to, an instructional video, a video showing thecompleted project, and/or a YouTube instructional video. The videos 34may have visually distinct display attributes (e.g., a different border,includes an overlay with a play button) from display attributes of theplurality of images 40. In some embodiments, when the user selects thevideo 34 from the cascading waterfall layout 18, the selected video 34may start playing in an overlay above the cascading waterfall layout 18nearby the display attributes for the selected video 34.

In some embodiments, the presentation component 28 may presentrecommendations 49 and/or advertising with the plurality of images 40and/or one or more videos 34 in the cascading waterfall layout 18. Therecommendations 49 may include similar content to the content expressedin the selected image and/or the content expressed in the associatedwebpage. The recommendations 49 may provide categories or suggestionsthat a user may select for additional ideas for the task 14 and/orsuggestions for different tasks 14. The recommendations 49 may havevisually distinct display attributes (e.g., different border, differentshading, overlays with the categories) from display attributes of thevideos 34 and/or the display attributes of the plurality of images 40.

In some embodiments, the presentation component 28 may presentadditional content 52 with the plurality of images 40 and/or one or morevideos 34 in the cascading waterfall layout 18. The additional content52 may include articles discussing the task 14. The additional content52 may have visually distinct display attributes (e.g., different borderor different shading) from display attributes of the videos 34 and/orthe display attributes of the plurality of images 40.

The search result webpage 16 may include one or more filter options 54for filtering the cascading waterfall layout 18 based on the structureddata 22 associated with each image of the plurality of images 40.Filters 54 may include, but are not limited to, cost, time, materials,categories, and/or ingredients. The presentation component 28 may adjustthe presentation of the cascading waterfall layout 18 based on the oneor more filters 54 selected. For example, the presentation component 28may remove one or more images from the plurality of images 40 based onthe selected filters 54. As such, the presentation may dynamicallyfilter the images 40 included in the cascading waterfall layout 18 basedon the selected filters 54.

The search result webpage 16 may present the cascading waterfall layout18 with the initial set of images of the plurality of images 40 and thesearch results 24. In some embodiments, the search results 24 arepresented below the cascading waterfall layout 18. As such, thecascading waterfall layout 18 is presented in a top portion of thesearch result webpage 16 and the search results 24 are presented in alower portion of the search result webpage 16.

In some embodiments, one or more computing devices (e.g., servers 106and/or device 102) is used to perform the processing of environment 100.The one or more computing devices may include, but are not limited to,server devices, personal computers, a mobile device, such as, a mobiletelephone, a smartphone, a PDA, a tablet, or a laptop, and/or anon-mobile device. The features and functionalities discussed herein inconnection with the various systems may be implemented on one computingdevice or across multiple computing devices. For example, the searchengine 26, the query classifier 29, and/or the presentation component 28is implemented wholly on the same computing device. Another exampleincludes one or more subcomponents of the search engine 26, the queryclassifier 29, and/or the presentation component 28 is implementedacross multiple computing devices. Moreover, in some embodiments, thesearch engine 26 and/or the query classifier 29 may be implemented areprocessed on different server devices of the same or different cloudcomputing networks.

In some embodiments, each of the components of the environment 100 is incommunication with each other using any suitable communicationtechnologies. In addition, while the components of the environment 100are shown to be separate, any of the components or subcomponents may becombined into fewer components, such as into a single component, ordivided into more components as may serve a particular embodiment. Insome embodiments, the components of the environment 100 includehardware, software, or both. For example, the components of theenvironment 100 may include one or more instructions stored on acomputer-readable storage medium and executable by processors of one ormore computing devices. When executed by the one or more processors, thecomputer-executable instructions of one or more computing devices canperform one or more methods described herein. In some embodiments, thecomponents of the environment 100 include hardware, such as a specialpurpose processing device to perform a certain function or group offunctions. In some embodiments, the components of the environment 100include a combination of computer-executable instructions and hardware.

As such, environment 100 may be used to visually represent search engineresults by providing a cascading waterfall layout 18 of search engineresults that is concise enough to allow quick visual comparison andexploration of the ideas presented in the cascading waterfall layout 18.Moreover, the user may quickly browse more ideas by reviewing thestructured data 22 for different images selected from the cascadingwaterfall layout 18 without having to redirect to the webpagesassociated with the selected images. As such, users may use environment100 to make more accurate and/or more informed decisions for determiningwhich webpages to visit provided in a search result.

Referring now to FIG. 2 , illustrated is an example graphical userinterface 200 displayed in a search result webpage 16 (FIG. 1 ) on adisplay 10 (FIG. 1 ) of a device 102 (FIG. 1 ). A user may enter in aquery 204 for “DIY mason jar” in the search box 202. The DIY project(DIY mason jar) may be a multi-step task. The query 204 may include arequest for details for completing the DIY project (DIY mason jar).

The presentation component 28 (FIG. 1 ) may provide a cascadingwaterfall layout 208 that includes a plurality of images 40 (FIG. 1 )from a plurality of webpages 38 (FIG. 1 ) that include details for DIYmason jar projects returned from a search engine 26 (FIG. 1 ). Thecascading waterfall layout 208 may correspond to the cascading waterfalllayout 18 discussed in FIG. 1 . The cascading waterfall layout 208 mayinclude four columns of images. The size of the plurality of imagesdisplayed in the cascading waterfall layout 208 may differ between rowsand/or columns. The columns may be a fixed size and the number ofcolumns may be selected based on a display size 48 (FIG. 1 ) of thedisplay 10. The cascading waterfall layout 208 includes three rows fordisplaying the plurality of images. The number of rows may be selectedbased on the display size 48 and/or the sizes of the images. If theplurality of images are landscape-oriented with a small height, morerows may be selected relative to images with a larger height. Inaddition, the size of the plurality of images displayed in the cascadingwaterfall layout 208 may change based on the display size 48. Thecascading waterfall layout 208 may be presented differently on devices102 with different display sizes 48 (e.g., a different number of columnsand/or rows may be presented). As such, the cascading waterfall layout208 has an adaptive arrangement of columns based on available screenreal estate, the size of the images, and/or responsive design.

A user may select an image 218 from the plurality of images in thecascading waterfall layout 208 to explore more. Adjacent to thecascading waterfall layout 208, structured data 210 for a selected image218 is presented. The structured data 210 includes an image 212 for “howto make a mason jar herb garden.” The image 212 provides additionaldetails and/or an overview of the DIY mason jar project discussed in thewebpage for the selected image 212. The structured data 210 alsoincludes a URL 214 to the webpage for the selected image 218. Thestructured data 210 also includes a plurality of images 216 showing stepby step details for completing the task for how to make a mason jar herbgarden.

The search engine 26 (FIG. 1 ) may provide one or more filter options206 for different DIY mason jar tasks. The filter options 206 mayinclude different categories (e.g., decor, Christmas, centerpieces,painted, ideas, projects, crafts, lanterns, snow globe, candles, gifts,and/or glitter) for DIY mason jar tasks. If the user clicks on any ofthe filter options 206, the presentation of the cascading waterfalllayout 208 may be adjusted to only include images associated with theselected filter option 206.

Referring now to FIG. 3 , illustrated is an example graphical userinterface 300 displayed in a search result webpage 16 (FIG. 1 ) on adisplay 10 (FIG. 1 ) of a device 102 (FIG. 1 ). A user may enter in aquery 12 (FIG. 1 ) for “DIY landscaping ideas” on a search webpage. TheDIY landscaping ideas may be a multi-step task. The query 12 may includea request for task details for completing the DIY project (DIYlandscaping ideas). The cascading waterfall layout 302 may include sixcolumns with four rows of images for the DIY project. The cascadingwaterfall layout 302 may be presented differently on devices 102 withdifferent display sizes 48 (e.g., a different number of columns and/orrows may be presented).

A user may select an image 310 from the plurality of images in thecascading waterfall layout 302 to explore more. Adjacent to thecascading waterfall layout 302, structured data 304 for the selectedimage 310 is presented. The structured data 304 includes an image 308for “10 Simple DIY Landscaping Ideas for Your Home on the Cheap.” Theimage 308 provides additional details and/or an overview of the DIYlandscaping project discussed in the webpage for the selected image 310.

Related content 306 to the selected image 308 may also be presented withthe structured data 304. The search engine 26 may provide relatedcontent 306 based on a combination of the data in the webpage associatedwith the selected image 310 and/or the selected image 310. For example,the selected image 310 is for “10 Simple DIY Landscaping Ideas for YourHome on the Cheap” and the related content 306 may be other low cost DIYprojects for your home. As such, the related content 306 provided in thestructured data 304 may be based on the information in the webpage ofthe selected image 310. Thus, as the user selects different images fromthe cascading waterfall layout 302, the related content 306 may changebased on the different content included in the webpages associated withthe newly selected images.

Referring now to FIGS. 4-7 , illustrated are examples of graphical userinterface 400 displayed in a search result webpage 16 (FIG. 1 ) on adisplay 10 (FIG. 1 ) of a device 102 (FIG. 1 ). A user may enter in aquery 404 for “strawberry cake recipes” in the search box 402. The query404 may include a request for details for preparing or cooking astrawberry cake.

The presentation component 28 (FIG. 1 ) may present on the search resultwebpage 16 a cascading waterfall layout 408 that includes a plurality ofimages 40 (FIG. 1 ) from a plurality of webpages 38 (FIG. 1 ) thatinclude details for strawberry cake recipes returned from a searchengine 26 (FIG. 1 ). The cascading waterfall layout 408 may correspondto the cascading waterfall layout 18 discussed in FIG. 1 . The cascadingwaterfall layout 408 may include seven columns of images. The size ofthe plurality of images displayed in the cascading waterfall layout 408may differ between rows and/or columns. The cascading waterfall layout408 includes two rows for displaying the plurality of images 40 and anicon 410 that may be selected to provide additional images in thecascading waterfall layout 408. The user may use the cascading waterfalllayout 408 to visually explore different recipes for strawberry cakes.The cascading waterfall layout 408 may be presented differently ondevices 102 with different display sizes 48 (e.g., a different number ofcolumns and/or rows may be presented).

The presentation component 28 may also present on the search resultwebpage 16 one or more filters options 406 for filtering the cascadingwaterfall layout 408. The filter options 406 include “top rated,”“quick,” “low calorie,” “low carbohydrates,” “low fat,” “high protein,”and “low sodium.” The user may select one or more of the filter options406 to only have images associated with the selected filter optionsshown on the cascading waterfall layout 408.

In addition, the presentation component 28 may present on the searchresult webpage 16 search results 24 provided from the search engine 26for the query 404 for “strawberry cake recipes.” The search results 412may be from a general search index and are presented below the cascadingwaterfall layout 408.

If the user selects the icon 410 to see additional images, the graphicaluser interface 400 is updated as illustrated in FIG. 5 . Referring nowto FIG. 5 , the cascading waterfall layout 408 expands to includeadditional rows of images in response to the user selecting the icon410. The number of rows increases from two rows to four rows, while thenumber of columns remain the same at seven. The one or more filteroptions 406 may remain on the search result webpage 16. However, theadditional search results 412 may no longer be visible on the searchresult webpage 16 without scrolling the search result webpage 16 upand/or down.

Referring now to FIG. 6A, if a user selects an image 604 from cascadingwaterfall layout 408, the structured data 602 associated with thewebpage for the selected image 604 is presented on the search resultwebpage 16. The presentation component 28 adjusts the presentation ofthe cascading waterfall layout 408 to make room for the structured data602. The presentation component 28 reduces the number of columns fromseven to five. In addition, the presentation component 28 rearranges thedisplayed images to fit within the rows.

The structured data 602 may include the ingredients 605 needed for thestrawberry cake recipe discussed in the webpage associated with theselected image 604. The structured data 602 may also include userratings and a URL link to the webpage. In addition, the structured data602 may include any ingredient substitutes that may be made to therecipe. The structured data 602 may also include an estimated time 607required to complete the recipe.

The images previously displayed in the cascading waterfall layout 408are still visible on the search result webpage 16 by scrolling down onthe search result webpage 16, as illustrated in FIG. 6B. Referring nowto FIG. 6B, if the user scrolls down to view additional images, aposition of the structure data 602 on the display 10 remains constantwhile the user is scrolling through the rows of the plurality of imagesin the cascading waterfall layout 408. Moreover, the informationprovided in the structured data 62 (e.g., the ingredients 605 needed forthe strawberry cake recipe, user ratings, a URL link to the webpage, anyingredient substitutes that may be made to the recipe, and/or anestimated time 607 required to complete the recipe) remains the same asthe user scrolls through additional rows of the plurality of images inthe cascading waterfall layout 408. The one or more filter options 406(FIG. 6A) may remain on the search result webpage 16.

Referring now to FIG. 7 , if a user selects a different image 702 fromcascading waterfall layout 408, the structured data 704 associated withthe webpage for the selected image 702 is presented on the search resultwebpage 16. The presentation component 28 may replace the previouslypresented structured data 602 (FIG. 6A) for the selected image 604 (FIG.6A) with the structured data 704 for the newly selected image 702. Theother images remain displayed in the cascading waterfall layout 408. Thedisplay attributes of the selected image 702 may be visually distinctfrom the remaining images displayed in the cascading waterfall layout408 (e.g., the selected image 702 is darker relative to the otherimages). As such, a persistent display of structured data may remain onthe search result webpage 16 as the user selects different imagespresented in the cascading waterfall layout 408.

The structured data 704 may include the ingredients needed for thestrawberry cake recipe discussed in the webpage associated with theselected image 702. The structured data 704 may also include userratings and a URL link to the webpage. In addition, the structured data704 may include nutrition facts for the recipe and/or any ingredientsubstitutes that may be made to the recipe.

Referring now to FIG. 8 , illustrated is an example graphical userinterface 800 displayed in a search result webpage 16 (FIG. 1 ) on adisplay 10 (FIG. 1 ) of a device 102 (FIG. 1 ). A user may enter in aquery 812 for “dinner recipes.” The query 812 may include a request forsuggestions and/or recommendations for dinner recipes.

The presentation component 28 (FIG. 1 ) may provide a cascadingwaterfall layout 806 to present on a search result webpage 16 (FIG. 1 )that includes a plurality of images 40 (FIG. 1 ) from a plurality ofwebpages 38 (FIG. 1 ) returned from a search engine 26 (FIG. 1 ) withinformation for different dinner recipes. The cascading waterfall layout806 includes five columns with two rows of images in the illustratedexample. The cascading waterfall layout 806 may be presented differentlyon devices 102 with different display sizes 48 (e.g., a different numberof columns and/or rows may be presented).

The presentation component 28 may also present on the search resultwebpage 16 one or more filters options 810 for filtering the cascadingwaterfall layout 806. The filter options 810 include “quick,” “easy,”“healthy,” “time,” “dietary restrictions,” “allergies,” “nutrition,”“techniques,” and “cuisine.” The user may select one or more of thefilter options 810 to only have images associated with the selectedfilter options shown on the cascading waterfall layout 806.

The one or more filter options 810 may include a drop down that allowsfurther selection by the user. If the user selects the filter optionallergies 802, a drop down menu 804 may be presented with differentselections the user may pick for allergies. For example, if the userselects gluten-free, the cascading waterfall layout 806 may be filteredto only show images of recipes that are gluten-free. The queryclassifier 29 may have performed one or more machine learning searcheson the webpages associated with images to determine whether the recipesare gluten-free. The machine learning searches may evaluate theingredients to determine if any of the ingredients have gluten and/orthe text of the webpage to determine whether the recipe is gluten free.For example, if a recipe has flour, the query classifier 29 determinesthat that recipe has gluten. However, if the recipe uses “rice flour,”the query classifier 29 may determine that the recipe is gluten-free.Another example may include the webpage including text that says, “easygluten-free recipes.” The query classifier 29 may use the text indetermining that the recipe is gluten-free.

Referring now to FIG. 9 , illustrated is an example graphical userinterface 900 displayed in a search result webpage 16 (FIG. 1 ) on adisplay 10 (FIG. 1 ) of a device 102 (FIG. 1 ). For example, a user mayenter in a general query 12 for cooking and the graphical user interface900 may include a cascading waterfall layout 902 with a plurality ofimages 40 (FIG. 1 ) from a plurality of webpages 38 (FIG. 1 ) associatedwith a tasks 14 for cooking. The cascading waterfall layout 902 may bepresented differently on devices 102 with different display sizes 48(e.g., a different number of columns and/or rows may be presented). Thecascading waterfall layout 902 may also include a plurality of videos904, 910, 912, 918 relating to cooking. The videos 904, 910, 912, 918optionally have visually distinct display attributes from the displayattributes of the plurality of images to indicate to a user that a videois displayed instead of an image. For example, the videos may include anoverlay (e.g., with a play button), a distinctive border, or texthighlighting.

The cascading waterfall layout 902 may also include a news article 906on cooking. In addition, the cascading waterfall layout 902 may alsoinclude one or more recommendations 916 for recipes for differentcooking categories (e.g., vegetarian recipes, dinner recipes, butternutsquash recipes, and/or Instant pot recipes for fall). Therecommendations 916 may be based on the original query or based on thecontent discussed in the plurality of webpages 38 included in thecascading waterfall layout 902. The recommendations 916 may includesimilar content to the content expressed in the plurality of images 40and/or the content expressed in the plurality of webpages 38. Therecommendations 916 may have visually distinct display attributes (e.g.,different backgrounds, different boarders) from the display attributesof the plurality of images to indicate to a user that one or morerecommendations 916 are provided instead of an image.

Referring now to FIG. 10 , illustrated is an example method 1000performed by device 102 (FIG. 1 ) for presenting structured data 22(FIG. 1 ) with search engine results 24 (FIG. 1 ) on a display 10 (FIG.1 ) of a device 102 (FIG. 1 ). The actions of method 1000 may bediscussed below with reference to the architecture of FIG. 1 .

At 1002, method 1000 may include receiving a query from a user for atask. The device 102 may receive a query 12 from a user of device 102.The user may enter queries 12 for searching different webpages for atask 14 and/or performing a visual exploration of one or more tasks 14.The query 12 may identify a multi-step task 14. In addition, the query12 may include a request for task details for completing the task. Forexample, the user may enter in the query 12 on a search engine webpage.

At 1004, method 1000 may include providing the query to a search engine.The device 102 may provide the query 12 to one or more search engines 26in communication with the device 102. The search engine 26 may searchone or more datastores 108, 110 with a plurality of webpages 31, 33 inresponse to the received query 12. The search engine 26 may generatesearch results 24 that identify a plurality of webpages 30 from thedatastores 108, 110 that discuss the task 14, include details tocomplete the task 14, and/or may be relevant to the task 14. The searchengine 26 may also generate textual search results 24 with a pluralityof search results for the query 12. The webpages 30 included in thesearch results 24 may include a plurality of images 32, a plurality ofvideos, and/or a plurality of text 36.

A query classifier 29 may perform further processing on the searchresults 24 to determine which images 32 and/or which webpages 30 fromthe search results 24 to present for the task 14 in response to thequery 12. The query classifier 29 may be a trained machine learningmodel that analyzes the search results 24 and the query 12 to determinewhich webpages 30 and/or images 32 of the search results 24 to presentfor the task 14. The segment or domain (e.g., recipes, DIY, travelplans) of the task 14 may be predicted by another query classifiertrained on human labeled data. The query classifier 29 may be trainedbased on fully supervised data (labeled by humans) or weakly superviseddata for more than one segments and/or domains. Different queryclassifiers 29 may be selected for analyzing the search results 24 basedon the segment or domain of the task 14.

The query classifier 29 may determine which images, videos, and/or gifsto include in the cascading waterfall by using a ranking 42 based on aweighted combination of aspects. For example, the image relevance to thetask, image attractiveness, and/or page usefulness may go into theranking. The results of the ranking 42 may be deduplicated based onimage content. For exploratory queries, higher rankings 42 may beassigned to page related aspects and idea-based deduplication may beapplied. For example, if there are multiple images for the same idea ofgarage decoration (e.g., just photographed from different angles), onlyone image per idea may be shown or the ideas may be grouped and shownwith multiple images with the same context information.

The query classifier 29 may identify a plurality of images 40 to presentand/or a plurality of webpages 38 to present from the search results 24.As such, a subset of the images 32 and/or the webpages 30 provided inthe search results 24 may be selected for presentation in response tothe query 12.

At 1006, method 1000 may include receiving search results from thesearch engine that include an identification of a plurality of webpageswith details to complete the task. The search engine 26 may send thesearch results 24 to the device 102 for presentation on the display 10.The search results 24 may be based on the general search index for thetask 14. The search engine 26 may also send to the device 102 theidentified plurality of webpages 38 that include details and/orinformation for completing the task 14. The information included in thesearch results 24 may be different from the information included in theidentified plurality of webpages 38.

At 1008, method 1000 may include presenting a search result webpageincluding at plurality of images from the plurality of webpages wherethe plurality of images are presented in a cascading waterfall layout.The device 102 may have a presentation component 28 that determines apresentation on the search result webpage 16 of the received searchresults 24, the plurality of webpages 38 and/or associated plurality ofimages 40, and/or any received structured data 22 for the plurality ofwebpages 38. The presentation component 28 may generate a cascadingwaterfall layout 18 to present the plurality of images 40 for theidentified plurality of webpages 38. The plurality of images 40 mayinclude, for example, images, gifs, and/or videos.

The presentation component 28 may determine to trigger the presentationof the cascading waterfall layout 18 based on the query 12 provided bythe user. The presentation component 28 may use a trained machinelearning model to determine an image intent in a query 12 and selectwhether the present the cascading waterfall layout 18 based on the imageintent of the query 12. As such, the presentation component 28 may usethe trained machine learning model to determine that the query 12 is avisual exploration search and may trigger the presentation of thecascading waterfall layout 18.

The cascading waterfall layout 18 may include a plurality of columns 50for presenting the plurality of images 40 from the plurality of webpages38. The width of the plurality of columns 50 may be based on the imageresult sizes and/or the display size 48 of the device 102 The width ofthe plurality of columns 50 may be fixed and the number of columnsincluded in the plurality of columns 50 may be based on the display size48 of the device 102. The number of rows displayed in the cascadingwaterfall layout 18 may be selected based on the display size 48 and/orthe size of the images. The presentation component 28 may place theplurality of images 40 into the plurality of columns 50 using therankings 42 and/or the presentation component 28 may use informationabout the presentation of the plurality of images 40 in the originalwebpages 30 in determining the order for presenting the plurality ofimages 40.

The presentation component 28 may start with a first image from theplurality of images 40 and place the image into a first column of theplurality of columns 50. A next image from the plurality of images 40 isplaced into a second column of the plurality of columns 50. The imageplacement continues until one image is placed per column. The images mayhave different sizes and the images per column may differ in size. Thepresentation component 28 may calculate which column has a least numberof pixels from the top of the search result webpage 16 and places a nextimage from the plurality of images 40 into the identified column. Thepresentation component 28 may continue to calculate which column has aleast number of pixels from the top of the search result webpage 16 andplace a next image into the identified column until a predeterminednumber of images are presented in the cascading waterfall layout 18. Thepredetermined number of images may present an initial subset of theplurality of images 40 to include in the cascading waterfall layout 18.As such, the presentation component 28 may fill up the cascadingwaterfall layout 18 with images based on a combination of the rankingsof the images 40 and the display requirements of the device 102. Theplurality of images 40 initially presented in the cascading waterfalllayout 18 may be the images with a highest ranking 42.

At 1010, method 1000 may include receiving a selection of an image fromthe plurality of images. A user may select one image of the plurality ofimages 40 presented in the cascading waterfall layout 18. For example,the user may want to review the image further or obtain additionalinformation for the selected image. The display attributes of theselected image may be visually distinct from the remaining imagesdisplayed in the cascading waterfall layout 18 (e.g., the selected imageis darker relative to the other images) to indicate which image isselected by the user.

At 1012, method 1000 may include presenting, adjacent to the pluralityof images on the search result webpage, structured data corresponding tothe image in response to the user selection. The presentation component28 may present the structured data 22 associated with the webpage forthe selected image. The structured data 22 may include, but is notlimited to, recipes, prices, user ratings, reviews, a uniform resourcelocator (URL) link to the webpage, the webpage name steps for performingthe task 14, step-by-step images for performing the task 14, aninstructional video for performing the task 14, a list of materialsneeded for performing the task 14, an estimated time required tocomplete the task 14, direct links to a portion of the webpage where thetask 14 is described, an aggregate of information from differentwebpages for the task 14 (additional images, steps, materialsdescription, etc.), substitute materials for the task 14, and/or localinformation 44 associated with the task 14.

The structured data 22 may be presented adjacent to the cascadingwaterfall layout 18. In some embodiments, the presentation component 28may present the structured data using a rich card 20. By presenting thestructured data 22 next to the cascading waterfall layout 18 on the samesearch result webpage 16, users may make more informed decisions aboutwhether to go to the source webpage for the selected image.

One example use case may include a query 12 for a DIY project forbuilding a bird house. The cascading waterfall layout 18 may include aplurality of images 40 for different DIY bird house building projects. Auser may select one of the images and the structured data 22 for thewebpage discussing the bird house DIY project shown in the image. Thestructured data 22 is presented on the side of the images in thecascading waterfall layout 18. The structured data 22 may include amaterials list for the bird house DIY project. In addition, thestructured data 22 may include step-by-step images for performing thebird house DIY project and an estimated time to complete the bird houseDIY project.

The structured data 22 may also include local information 44 providedfrom sources other than the webpage associated with the selected image.The local information 44 may include a local cost of materials for thebird house DIY project, local availability of the materials, and a storelocated nearby the device 102 where the user may purchase the materials.As such, the user may use the local information 44 to determine the costof the bird house DIY project and where to purchase the materialslocally.

One or more recommendations 49 and/or advertisements may be presentedwith the structured data 22 based on the selected image. Therecommendations 49 may include similar content to the content expressedin the selected image and/or the content expressed in the associatedwebpage. The recommendations 49 may include tools needed for theproject. Even if the webpage does not mention tools needed to completethe project, the tools recommended may be based on tools needed forsimilar projects. If the materials cost $25, the recommendations 49 mayinclude other DIY projects for $25. If the user selects a differentimage of a bird house DIY project where the materials cost $100, therecommendations 49 may include other DIY projects for $100. As such, theDIY projects included in the recommendations 49 may change based on theselected images and the information associated with the webpage for theimage without modifying the plurality of images 40 included in thecascading waterfall layout 18.

Another example use case may include a query 12 for healthy lasagnarecipes. The cascading waterfall layout 18 may include a plurality ofimages 40 for associated source webpages that discuss different healthyrecipes for lasagna. A user may select one of the images and thestructured data 22 for the selected image may be presented next to thecascading waterfall layout 18. The structured data 22 may include alisting of ingredients for the recipe from the associated webpage. Inaddition, the structured data 22 may include a direct link to a portionof the webpage where the recipe is located. For example, if the userselects the direct link, the webpage may open directly to the portion ofthe webpage where the recipe is included. As such, the user may save afew seconds from having to scroll through the webpage to locate thestart of the recipe. The structured data 22 may also include aninstruction section that provides the users a few sentences for how tomake the recipe.

The structured data 22 may also include local information 44 providedfrom sources other than the webpage associated with the selected image.The local information 44 may include a local cost of ingredients for therecipe, local availability of the ingredients, and a store locatednearby the device 102 where the user may purchase the ingredients. Assuch, the user may use the local information 44 to determine cost of theingredients for the recipe and where to purchase the ingredientslocally.

One or more recommendations 49 may be presented with the structured data22 based on the selected image. The recommendations 49 may includesimilar content to the content expressed in the selected image and/orthe content expressed in the associated webpage. For example, otherrecipes may be presented with the structured data 22 based on theselected image. If the selected image is for lasagna soup, therecommendations 49 may include other healthy soup recipes. If theselected image is for chocolate lasagna, the recommendations 49 mayinclude other healthy dessert recipes. If the selected image is forlasagna, the recommendations 49 may include other pasta ideas. Therecommendations 49 may include images of the recommended recipes. Therecommendations 49 may not be included in the cascading waterfall layout18 but may be obtained in response to a new query constructed based onthe selected image. As such, the recipes included in the recommendations49 may change based on the selected images without modifying theplurality of images 40 included in the cascading waterfall layout 18.

Another example use case may include a query 12 for a trip to Paris,France. The cascading waterfall layout 18 may include a plurality ofimages 40 for associated source webpages that discuss travelling toParis. A user may select one of the images and the structured data 22associated with the selected image may be presented adjacent to thecascading waterfall layout 18. The structured data 22 may include anaggregation of information from multiple webpages. For example, thestructured data 22 may include user reviews from a plurality of webpagesof the top ten restaurants in Paris. Another example may include thestructure data 22 may include the top twenty attractions in Paris withhighlights from each attraction provided.

At 1014, method 1000 may include adjusting the presentation of thecascading waterfall layout based on the selection of the image. Thepresentation component 28 adjusts the presentation of the cascadingwaterfall layout 18 based on the selection of the image so that thestructured data 22 is presented next to the cascading waterfall layout18. Adjusting the presentation of the cascading waterfall layout 18 mayinclude, for example, resizing a column of the plurality of columns 50,reducing a number of columns of the plurality of columns 50, rearrangingan ordering of the images within at least one column of the plurality ofcolumns 50, resizing the images within the plurality of columns 50,and/or rearranging a position of the images within the plurality ofcolumns 50. As such, the cascading waterfall layout 18 has an adaptivearrangement of columns based on available screen real estate andresponsive design.

The presentation component 28 adjusts the presentation of the cascadingwaterfall layout 18 without issuing a new search request to the searchengine 26 and/or changing the query 12 from the user. If a user selectsa different image from the plurality of images 40, the presentationcomponent 28 may replace the structured data 22 presented with thestructured data 22 for the webpage associated with the different image.The position of the structured data 22 on the display 10 remainsconstant while the user is scrolling through rows of the plurality ofimages in the cascading waterfall layout 18 and/or selecting differentimages to view.

The user may view different structure data 22 for different imagesselected from the plurality of images 40 without issuing a new search tothe search engine 26 or changing the query 12 from the user, resultingin an improvement of the search result webpage 16 by not having torefresh the entire search result webpage 16 each time a new image isselected. As such, the user may easily switch between the structureddata 22 for different images selected resulting in quick and easycomparison of the different tasks 14 for the selected images. Moreover,the user may quickly browse more ideas by reviewing the structured data22 for different images selected without having to redirect to thewebpages associated with the selected images.

The search result webpage 16 may include one or more filter options 54for filtering the cascading waterfall layout 18 based on the structureddata 22 associated with each image of the plurality of images 40.Filters 54 may include, but are not limited to, cost, time, materials,categories, and/or ingredients. The presentation component 28 may adjustthe presentation of the cascading waterfall layout 18 based on the oneor more filters 54 selected. For example, the presentation component 28may remove one or more images from the plurality of images 40 based onthe selected filters 54. As such, the presentation may dynamicallyfilter the images 40 included in the cascading waterfall layout 18 basedon the selected filters 54.

The search result webpage 16 may present the cascading waterfall layout18 and the search results 24. In some embodiments, the search results 24are presented below the cascading waterfall layout 18. As such, thecascading waterfall layout 18 is presented in a top portion of thesearch result webpage 16 and the search results 24 are presented in alower portion of the search result webpage 16.

Another example method performed by a server 106 (FIG. 1 ) may includegenerating and returning search results 24 (FIG. 1 ) for a query 12(FIG. 1 ) to a device 102 (FIG. 1 ). The actions of method may bediscussed below with reference to the architecture of FIG. 1 . Themethod may include receiving a query 12 for a task 14. The search engine26 may receive one or more queries 12 for a task 14 from one or moredevices 102.

The method may include generating a task index for the query andquerying one or more datastores for webpages with information related tothe task. The search engine 26 may generate a task index for the query12 and may query one or more datastores 108, 110 with webpages 31, 33.The search engine 26 may generate search results 24 with anidentification of a plurality of webpages, wherein each webpage of theplurality of webpages includes details to complete the task 14.

The method may include supplementing the search results with third partydata for the task. For example, the search engine 26 may supplement thesearch results 24 with local information 44 provided by a third partydatastore 112. The local information 44 may include, but is not limitedto, details about the price and/or availability of materials for thetask 14 based on the location of the device 102.

The method may include returning the search results and the third partydata for the task to the device. The search engine 26 may send thesearch results 24 and the local information 44 to the device 102 inresponse to receiving the query 12.

As illustrated in the foregoing discussion, the present disclosureutilizes a variety of terms to describe features and advantages of thequery classifier, the search engine, and/or the presentation component.Additional detail is now provided regarding the meaning of such terms.For example, as used herein, a “machine learning model” refers to acomputer algorithm or model (e.g., a classification model, a regressionmodel, a language model, an object detection model) that can be tuned(e.g., trained) based on training input to approximate unknownfunctions. For example, a machine learning model may refer to a neuralnetwork (e.g., a convolutional neural network (CNN), deep neural network(DNN), recurrent neural network (RNN)), or other machine learningalgorithm or architecture that learns and approximates complex functionsand generates outputs based on a plurality of inputs provided to themachine learning model. As used herein, a “machine learning system” mayrefer to one or multiple machine learning models that cooperativelygenerate one or more outputs based on corresponding inputs. For example,a machine learning system may refer to any system architecture havingmultiple discrete machine learning components that consider differentkinds of information or inputs.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof, unless specificallydescribed as being implemented in a specific manner. Any featuresdescribed as modules, components, or the like may also be implementedtogether in an integrated logic device or separately as discrete butinteroperable logic devices. If implemented in software, the techniquesmay be realized at least in part by a non-transitory processor-readablestorage medium comprising instructions that, when executed by at leastone processor, perform one or more of the methods described herein. Theinstructions may be organized into routines, programs, objects,components, data structures, etc., which may perform particular tasksand/or implement particular data types, and which may be combined ordistributed as desired in various embodiments.

Computer-readable mediums may be any available media that can beaccessed by a general purpose or special purpose computer system.Computer-readable mediums that store computer-executable instructionsare non-transitory computer-readable storage media (devices).Computer-readable mediums that carry computer-executable instructionsare transmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable mediums: non-transitorycomputer-readable storage media (devices) and transmission media.

As used herein, non-transitory computer-readable storage mediums(devices) may include RAM, ROM, EEPROM, CD-ROM, solid state drives(“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory(“PCM”), other types of memory, other optical disk storage, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer.

The steps and/or actions of the methods described herein may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

The term “determining” encompasses a wide variety of actions and,therefore, “determining” can include calculating, computing, processing,deriving, investigating, looking up (e.g., looking up in a table, adatabase, a datastore, or another data structure), ascertaining and thelike. Also, “determining” can include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Also, “determining” can include resolving, selecting, choosing,establishing and the like.

The articles “a,” “an,” and “the” are intended to mean that there areone or more of the elements in the preceding descriptions. The terms“comprising,” “including,” and “having” are intended to be inclusive andmean that there may be additional elements other than the listedelements. Additionally, it should be understood that references to “oneembodiment” or “an embodiment” of the present disclosure are notintended to be interpreted as excluding the existence of additionalembodiments that also incorporate the recited features. For example, anyelement described in relation to an embodiment herein may be combinablewith any element of any other embodiment described herein. Numbers,percentages, ratios, or other values stated herein are intended toinclude that value, and also other values that are “about” or“approximately” the stated value, as would be appreciated by one ofordinary skill in the art encompassed by embodiments of the presentdisclosure. A stated value should therefore be interpreted broadlyenough to encompass values that are at least close enough to the statedvalue to perform a desired function or achieve a desired result. Thestated values include at least the variation to be expected in asuitable manufacturing or production process, and may include valuesthat are within 5%, within 1%, within 0.1%, or within 0.01% of a statedvalue.

A person having ordinary skill in the art should realize in view of thepresent disclosure that equivalent constructions do not depart from thespirit and scope of the present disclosure, and that various changes,substitutions, and alterations may be made to embodiments disclosedherein without departing from the spirit and scope of the presentdisclosure. Equivalent constructions, including functional“means-plus-function” clauses are intended to cover the structuresdescribed herein as performing the recited function, including bothstructural equivalents that operate in the same manner, and equivalentstructures that provide the same function. It is the express intentionof the applicant not to invoke means-plus-function or other functionalclaiming for any claim except for those in which the words ‘means for’appear together with an associated function. Each addition, deletion,and modification to the embodiments that falls within the meaning andscope of the claims is to be embraced by the claims.

INDUSTRIAL APPLICABILITY

The present disclosure relates to systems, devices, and methods forpresenting structured data with search engine results on a display of adevice. This disclosure provides better context for comparing ideas(recipes, travel destinations, DIY projects, etc.) by showing not onlyimages, gifs, and/or videos for search results for a task, but alsorelevant page information using structured data extracted from thewebpage associated with the images. The structured data may include, butis not limited to, title of the webpage, recipes, prices, user ratings,reviews, a uniform resource locator (URL) link to the webpage, thewebpage name steps for performing the task 14, step-by-step images forperforming the task 14, an instructional video for performing the task14, a list of materials needed for performing the task 14, an estimatedtime required to complete the task 14, direct links to a portion of thewebpage where the task 14 is described, an aggregate of information fromdifferent webpages for the task 14 (additional images, steps, materialsdescription, etc.), substitute materials for the task 14, localinformation 44 associated with the task 14, and/or rich metadataextracted from the webpages.

The presentation provided by the methods and devices may include acascading waterfall layout with a plurality of images. The size of thecascading waterfall layout (e.g., the number of columns and/or thenumber of rows included) is based on the display size and/or the sizesof the images. The plurality of images may be associated with aplurality of webpages that include details for completing the task. Theplurality of webpages may be included in search results returned by asearch engine in response to a query for the task. The cascadingwaterfall layout is concise enough to allow quick comparison andexploration of the ideas. The cascading waterfall layout may bepresented at a top portion of a search result webpage above the generalsearch results returned by the search engine for a query.

A determination is made to trigger the cascading waterfall layout ofimages, videos, and/or gifs based on the query provided by the user. Atrained machine learning model may be used to determine whether totrigger the presentation of the cascading waterfall layout. The machinelearning model may be trained based on fully supervised data (labeled byhumans) or by weakly supervised data of search engine logs to extractimages searches where users are interested in the source pages, not onlyin the images, based on whether the user clicks on pages more often thanthe image from the search engine logs. The user clicking on the sourcepages may indicate that the query has visual exploration intent(exploration because of clicking on the webpages may indicate interestin additional context and/or related information) and visual intentbecause the query was issued to an image search, not a web search. Ifthe machine learning model determines that the query includes a visualintent, the presentation of the cascading waterfall layout of images maybe triggered.

A determination of which images, videos, and/or gifs to include in thecascading waterfall may be made by using a ranking based on a weightedcombination of aspects. For example, the image relevance to the task,image attractiveness, and/or page usefulness may go into the ranking.The results of the ranking may be deduplicated based on image content.For exploratory queries, higher weights may be assigned to page relatedaspects and idea-based deduplication may be applied. For example, ifthere are multiple images for the same idea of garage decoration (e.g.,just photographed from different angles), only one image per idea may beshown or the ideas may be grouped and shown with multiple images withthe same context information. The order of images presented in thecascading waterfall may be also specific (e.g., for DIY projects theorder of images presented may be a logical order of steps to completethe project). The methods and devices may also integrate a visual searchwhere users search by images (e.g., exploring ideas with similar look,such as, bedrooms in similar styles, colors) in determining which imagesto include in the cascading waterfall layout.

One example use case may include a query for a DIY project for buildinga bird house. The cascading waterfall layout may include a plurality ofimages for different DIY bird house building projects. A user may selectone of the images and the structured data for the webpage discussing thebird house DIY project shown in the image. The structured data ispresented on the side of the images in the cascading waterfall layout.The structured data may include a materials list for the bird house DIYproject. In addition, the structured data may include step-by-stepimages for performing the bird house DIY project and an estimated timeto complete the bird house DIY project.

The structured data may also include local information provided fromsources other than the webpage associated with the selected image. Thelocal information may include a local cost of materials for the birdhouse DIY project, local availability of the materials, and a storelocated nearby the device where the user may purchase the materials. Assuch, the user may use the local information to determine the cost ofthe bird house DIY project and where to purchase the materials locally.

One or more recommendations may be presented with the structured databased on the selected image. The recommendations may include similarcontent to the content expressed in the selected image and/or thecontent expressed in the associated webpage. The recommendations mayinclude tools needed for the project. Even if the webpage does notmention tools needed to complete the project, the tools recommended maybe based on tools needed for similar projects. If the materials cost$25, the recommendations may include other DIY projects for $25. If theuser selects a different image of a bird house DIY project where thematerials cost $100, the recommendations may include other DIY projectsfor $100. As such, the DIY projects included in the recommendations maychange based on the selected images and the information associated withthe webpage for the image without modifying the plurality of imagesincluded in the cascading waterfall layout.

Another example use case may include a query for healthy lasagnarecipes. The cascading waterfall layout may include a plurality ofimages for associated source webpages that discuss different healthyrecipes for lasagna. A user may select one of the images and thestructured data for the selected image may be presented next to thecascading waterfall layout. The structured data may include a listing ofingredients for the recipe from the associated webpage. In addition, thestructured data may include a direct link to a portion of the webpagewhere the recipe is located. For example, if the user selects the directlink, the webpage may open directly to the portion of the webpage wherethe recipe is included. As such, the user may save a few seconds fromhaving to scroll through the webpage to locate the start of the recipe.The structured data may also include an instruction section thatprovides the users a few sentences for how to make the recipe.

The structured data may also include local information provided fromsources other than the webpage associated with the selected image. Thelocal information may include a local cost of ingredients for therecipe, local availability of the ingredients, and a store locatednearby the device where the user may purchase the ingredients. As such,the user may use the local information to determine cost of theingredients for the recipe and where to purchase the ingredientslocally.

One or more recommendations may be presented with the structured databased on the selected image. The recommendations may include similarcontent to the content expressed in the selected image and/or thecontent expressed in the associated webpage. For example, other recipesmay be presented with the structured data based on the selected image.If the selected image is for lasagna soup, the recommendations mayinclude other healthy soup recipes. If the selected image is forchocolate lasagna, the recommendations may include other healthy dessertrecipes. If the selected image is for lasagna, the recommendations mayinclude other pasta ideas. The recommendations may include images of therecommended recipes. The recommendations may not be included in thecascading waterfall layout but may be obtained in response to a newquery constructed based on the selected image. As such, the recipesincluded in the recommendations may change based on the selected imageswithout modifying the plurality of images included in the cascadingwaterfall layout.

Another example use case may include a query for a trip to Paris,France. The cascading waterfall layout may include a plurality of imagesfor associated source webpages that discuss travelling to Paris. A usermay select one of the images and the structured data associated with theselected image may be presented adjacent to the cascading waterfalllayout. The structured data may include an aggregation of informationfrom multiple webpages. For example, the structured data may includeuser reviews from a plurality of webpages of the top ten restaurants inParis. Another example may include the structure data may include thetop twenty attractions in Paris with highlights from each attractionprovided.

Technical advantages of the systems, devices, and methods include theuser may view different structure data for different images selectedfrom the plurality of images within the cascading waterfall layoutwithout issuing a new search to the search engine or changing the queryfrom the user, resulting in an improvement of the search result webpageby not having to refresh the entire search result webpage each time anew image is selected. As such, the user may easily switch between thestructured data for different images selected resulting in quick andeasy comparison of the different tasks for the selected images.Moreover, the user may quickly browse more ideas by reviewing thestructured data for different images selected without having to redirectto the webpages associated with the selected images.

(A1) In one aspect, some embodiments include a method (1000) forpresenting structured data with search engine results on a display of adevice. The method includes receiving (1002) a query from a user (e.g.,query 204), where the query identifies a multi-step task (e.g., task 14)and comprises a request for task details for completing the task. Themethod also includes providing (1004) the query to a search engine(e.g., search engine 26). The method further includes, responsive toproviding the query, receiving (1006) results from the search engine,where the received results include identification of a plurality ofwebpages (e.g., webpages 38), where each webpage of the plurality ofwebpages includes details to complete the task. The method includespresenting (1008) on the display a search result webpage responsive toreceiving the results (e.g., search result webpage 16), the searchresult webpage including a plurality of images (e.g., images 40) fromthe plurality of webpages, where the plurality of images are presentedin a cascading waterfall layout that includes a plurality of columns(e.g., layout 208). The method further includes receiving (1010) a userselection of an image from the plurality of images; and, responsive tothe user selection: presenting (1012), adjacent to the plurality ofimages on the search result webpage, structured data corresponding tothe image (e.g., structured data 210), the structured data comprisingdetails from the webpage to complete the task; and adjusting (1014) thepresentation of the cascading waterfall layout based on the selection ofthe image, where the adjusting includes at least one of: resizing acolumn of the plurality of columns, and rearranging an ordering ofimages within at least one column of the plurality of columns.

(A2) In some embodiments of the method of A1, adjusting the presentationof the cascading waterfall layout occurs without issuing a new searchrequest to the search engine or changing the query from the user. Forexample, the structured data associated with each image in the pluralityof images is provided to the user device with the search result webpageand is displayed or hidden locally in response to user image selections.In this way the user may explore the images and structured data withoutsending additional search queries to the search engine.

(A3) In some embodiments of the method of A1 or A2, a number of columnsselected for the cascading waterfall layout is based on a size of thedisplay, and adjusting the presentation of the cascading waterfalllayout further includes reducing the number of columns displayed andrearranging a position of the plurality of images to fit within thereduced number of columns (e.g., as illustrated in FIGS. 5 & 6 ).

(A4) In some embodiments of the method of any of A1-A3, displaying theplurality of images further comprises: placing one image of theplurality of images into each column of the plurality of columns;determining which column has a least number of pixels from a top of thesearch result webpage; placing a next image of the plurality of imagesinto the column with the least number of pixels; and continuing todetermine which column has the least number of pixels from the top ofthe search result webpage and placing another image of the plurality ofimages into the column with the least number of pixels until apredetermined number of images are displayed in the cascading waterfalllayout. For example, in this way the image results located closest tothe top of the waterfall are the highest ranked and the image resultslocated lower on the waterfall are lower ranked.

(A5) In some embodiments of the method of any of A1-A4, the structureddata is presented at a position on the display, and the position of thestructured data on the display remains constant while a user isscrolling through rows of the plurality of images in the cascadingwaterfall layout (e.g., as illustrated in FIG. 6B).

(A6) In some embodiments of the method of any of A1-A5, the cascadingwaterfall layout is displayed at a top portion of the search resultwebpage and textual search results for the query (e.g., results 412) arepresented below the cascading waterfall layout.

(A7) In some embodiments of the method of any of A1-A6, at least onewebpage of the plurality of webpages includes a video describing taskdetails (e.g., video 904); and the cascading waterfall layout includesthe plurality of images and the video, where at least one displayattribute of the video is visually distinct from display attributes ofthe plurality of images.

(A8) In some embodiments of the method of any of A1-A7, the structuredata includes one or more of the following from the webpage: steps forperforming the task, step-by-step images for performing the task (e.g.,images 216), a list of materials needed for performing the task (e.g.,ingredients 605), an estimated time required to complete the task (e.g.,time 607).

(A9) In some embodiments of the method of any of A1-A8, the structureddata includes one or more of the following from sources other than thewebpage: estimated cost of materials for the task, local availability ofmaterials for the task, substitute materials for the task, or anaggregation of information from different webpages for the task. Forexample, the local availability of materials and/or estimated cost maycome from a local information database (e.g., local information 44 indatastore 112). As another example, the substitute materials may comefrom a third party materials datastore.

(A10) In some embodiments of the method of any of A1-A9, the searchresults webpage includes filter options (e.g., filters 54) for filteringthe cascading waterfall based on the structured data associated witheach image.

In another aspect, some embodiments include a device (102) having one ormore processors; memory in electronic communication with the one or moreprocessors; and instructions stored in the memory, the instructionsbeing executable by the one or more processors to perform any of themethods described herein (e.g., methods A1-A10).

In another aspect, some embodiments include a computer-readable mediumstoring instructions executable by a computer device (102). Thecomputer-readable medium may include at least one instruction forcausing the computer device (102) to perform any of the methodsdescribed herein (e.g., methods A1-A10).

What is claimed is:
 1. A method for presenting structured data with search engine results on a display of a device, comprising: receiving a query from a user, wherein the query identifies a multi-step task and comprises a request for task details for completing the task; providing the query to a search engine; responsive to providing the query, receiving results from the search engine, wherein the received results include identification of a plurality of webpages, wherein each webpage of the plurality of webpages includes details to complete the task; presenting on the display a search result webpage responsive to receiving the results, the search result webpage including a plurality of images from the plurality of webpages, wherein the plurality of images are presented in a cascading waterfall layout that includes a plurality of columns by placing one image of the plurality of images into each column of the plurality of columns and continuing to place images into each column based on an available number of pixels in a column until a predetermined number of images are displayed in the cascading waterfall layout; receiving a user selection of an image from the plurality of images; and responsive to the user selection: presenting, adjacent to the plurality of images on the search result webpage, structured data corresponding to the image, the structured data comprising details from the webpage to complete the task; and adjusting the presentation of the cascading waterfall layout based on the selection of the image, wherein the adjusting includes at least one of: resizing a column of the plurality of columns, and rearranging an ordering of images within at least one column of the plurality of columns.
 2. The method of claim 1, wherein adjusting the presentation of the cascading waterfall layout occurs without issuing a new search request to the search engine or changing the query from the user.
 3. The method of claim 1, wherein a number of columns selected for the cascading waterfall layout is based on a size of the display; and wherein adjusting the presentation of the cascading waterfall layout further includes reducing the number of columns displayed and rearranging a position of the plurality of images to fit within the reduced number of columns.
 4. The method of claim 1, wherein displaying the plurality of images further comprises: determining which column has a least number of pixels from a top of the search result webpage; placing a next image of the plurality of images into the column with the least number of pixels; and continuing to determine which column has the least number of pixels from the top of the search result webpage and placing another image of the plurality of images into the column with the least number of pixels until a predetermined number of images are displayed in the cascading waterfall layout.
 5. The method of claim 1, wherein the structured data is presented at a position on the display, and wherein the position of the structured data on the display remains constant while a user is scrolling through rows of the plurality of images in the cascading waterfall layout.
 6. The method of claim 1, wherein the cascading waterfall layout is displayed at a top portion of the search result webpage and textual search results for the query are presented below the cascading waterfall layout.
 7. The method of claim 1, wherein at least one webpage of the plurality of webpages includes a video describing task details; and wherein the cascading waterfall layout includes the plurality of images and the video, wherein at least one display attribute of the video is visually distinct from display attributes of the plurality of images.
 8. The method of claim 1, wherein the structure data includes one or more of the following from the webpage: steps for performing the task, step-by-step images for performing the task, a list of materials needed for performing the task, an estimated time required to complete the task.
 9. The method of claim 8, wherein the structured data includes one or more of the following from sources other than the webpage: estimated cost of materials for the task, local availability of materials for the task, substitute materials for the task, or an aggregation of information from different webpages for the task.
 10. The method of claim 1, wherein the search result webpage includes filter options for filtering the cascading waterfall based on the structured data associated with each image.
 11. A device, comprising: one or more processors; memory in electronic communication with the one or more processors; and instructions stored in the memory, the instructions executable by the one or more processors to: receiving a query from a user, wherein the query identifies a multi-step task and comprises a request for task details for completing the task; providing the query to a search engine; responsive to providing the query, receiving results from the search engine, wherein the received results include identification of a plurality of webpages, wherein each webpage of the plurality of webpages includes details to complete the task; presenting on a display a search result webpage responsive to receiving the results, the search result webpage including a plurality of images from the plurality of webpages, wherein the plurality of images are presented in a cascading waterfall layout that includes a plurality of columns by placing one image of the plurality of images into each column of the plurality of columns and continuing to place images into each column based on an available number of pixels in a column until a predetermined number of images are displayed in the cascading waterfall layout; receiving a user selection of an image from the plurality of images; and responsive to the user selection: presenting, adjacent to the plurality of images on the search result webpage, structured data corresponding to the image, the structured data comprising details from the webpage to complete the task; and adjusting the presentation of the cascading waterfall layout based on the selection of the image, wherein the adjusting includes at least one of: resizing a column of the plurality of columns, and rearranging an ordering of images within at least one column of the plurality of columns.
 12. The device of claim 11, wherein the instructions are further executable by the one or more processors to adjust the presentation of the cascading waterfall layout without issuing a new search request to the search engine or changing the query from the user.
 13. The device of claim 11, wherein the instructions are further executable by the one or more processors to select a number of columns for the cascading waterfall layout based on a size of the display; and adjust the presentation of the cascading waterfall layout by reducing the number of columns displayed and rearranging a position of the plurality of images to fit within the reduced number of columns.
 14. The device of claim 11, wherein the instructions are further executable by the one or more processors to display the plurality of images by: determining which column has a least number of pixels from a top of the search result webpage; placing a next image of the plurality of images into the column with the least number of pixels; and continuing to determine which column has the least number of pixels from the top of the search result webpage and placing another image of the plurality of images into the column with the least number of pixels until a predetermined number of images are displayed in the cascading waterfall layout.
 15. The device of claim 11, wherein the instructions are further executable by the one or more processors to present the structured data at a position on the display, and wherein the position of the structured data on the display remains constant while a user is scrolling through rows of the plurality of images in the cascading waterfall layout.
 16. The device of claim 11, wherein at least one webpage of the plurality of webpages includes a video describing task details; and wherein the cascading waterfall layout includes the plurality of images and the video, wherein at least one display attribute of the video is visually distinct from display attributes of the plurality of images.
 17. The device of claim 11, wherein the structure data includes one or more of the following from the webpage: steps for performing the task, step-by-step images for performing the task, a list of materials needed for performing the task, an estimated time required to complete the task.
 18. The device of claim 17, wherein the structured data includes one or more of the following from sources other than the webpage: estimated cost of materials for the task, local availability of materials for the task, substitute materials for the task, or an aggregation of information from different webpages for the task.
 19. The device of claim 11, wherein the search result webpage includes filter options for filtering the cascading waterfall based on the structured data associated with each image.
 20. A non-transitory computer-readable medium storing instructions executable by a computer device, comprising: at least one instruction for causing the computer device to receive a query from a user, wherein the query identifies a multi-step task and comprises a request for task details for completing the task; at least one instruction for causing the computer device to provide the query to a search engine; at least one instruction for causing the computer device to receive results from the search engine, responsive to providing the query, wherein the received results include identification of a plurality of webpages, wherein each webpage of the plurality of webpages includes details to complete the task; at least one instruction for causing the computer device to present on a display a search result webpage responsive to receiving the results, the search result webpage including a plurality of images from the plurality of webpages, wherein the plurality of images are presented in a cascading waterfall layout that includes a plurality of columns by placing one image of the plurality of images into each column of the plurality of columns and continuing to place images into each column based on an available number of pixels in a column until a predetermined number of images are displayed in the cascading waterfall layout; at least one instruction for causing the computer device to receive a user selection of an image from the plurality of images; and responsive to the user selection: at least one instruction for causing the computer device to present, adjacent to the plurality of images on the search result webpage, structured data corresponding to the image, the structured data comprising details from the webpage to complete the task; and at least one instruction for causing the computer device to adjust the presentation of the cascading waterfall layout based on the selection of the image, wherein the adjusting includes at least one of: resizing a column of the plurality of columns, and rearranging an ordering of images within at least one column of the plurality of columns. 